There is a reproduciblity crisis in academia

Back in 2005 a ground breaking paper by John Ioannidis¹ exposed an unsettling truth

There’s still a crisis almost two decades later

Despite lots being written about the crisis… it’s still here. And with the rise of

“machine learning can solve anything!”

… the crisis is evolving and getting more complicated, as reported by Douglas Heaven²

Reproducibility vs Replicability

Quite a lot of the time you’ll see these two words used interchangeably - but they have specific meanings. The American Statistical Association (ASA) provides useful advice from Broman et al³.

Reproducibility: A study is reproducible if you can take the original data and the computer code used to analyze the data and reproduce all of the numerical findings from the study.

• Your module assessment must be reproducible - I must be able to run the code after it’s submitted.

Reproducibility

Reproducibility: A study is reproducible if you can take the original data and the computer code used to analyze the data and reproduce all of the numerical findings from the study.

Reproducibility is something to think about from the start of a research project:

• Plan to record and document all processes in data collection, wrangling and analysis

• While performing the research keep track of all the things you do - particularly with data!

• When writing up your research ensure that all necessary materials to reproduce your findings are made available.

Later in this lecture and the workshop we’ll look at specific advice for achieving this.

Replicability

Replicability: This is the act of repeating an entire study, independently of the original investigator without the use of original data (but generally using the same methods)

Questionable Research Practices

There are lots of different ways to summarise common QRPs but I quite like this table from 2012 by John et al⁵.

Questionable Research Practices and P-hacking

The majority of these QRPs can be categorised as “P-hacking” or more fully - hacking the P-value.

What is the p-value?

We’re going to talk about p-values A LOT in week 10. For now I wanted to borrow a slide from Lucy D’Agostino McGowan’s talk which is well explained in this Twitter thread.

How folks treat the p-value

P-hacking

In 2019 Aschwanden⁶ published an article in Wired.com titled “We’re All ‘P-Hacking’ Now” which I highly recommend reading.

It highlights an excellent study by Simmons et al⁷ that was able to use p-value hacking to make two increasingly absurd conclusions:

Study 1: Listening to a children’s song (“Hot Potato” by The Wiggles) makes people feel older.

Study 2: Listening to a song about old age (“When I’m Sixty-Four” by The Beatles) makes people actually younger.

… I recommend this paper because it makes very clear recommendations to researchers and reviewers.

Reading papers is a skill (I)

Always expect to read a paper multiple times and make notes.

We can kind of neatly split papers into two different types:

Reading papers is a skill (II)

Reading papers is a skill (III)

Reading papers is a skill (IV)

If you decide to go into a research career you’ll likely be reading 10+ papers a week.

I’d highly recommend investing in learning speed reading early in your career.

There’s lots of [very interesting] eye tracking and neurological research into how we read and evidenced methodologies for speed reading that are nicely summarised by Clifton et al⁹.

Fighting p-hacking with pre-registration

So far I’ve been speaking about the academic literature at large.

But let’s look specifically at clinical trials.

All clinical trials began after July 1st 2005 are explicitly required to be registered¹ in order to be published in all biomedical journals overseen by the International Committee of Medical Journal Editors (ICMJE)¹⁰.

This is independent of the country in which the trial took place.

National Institutes of Health (NIH) and registration

From a cursory search for clinical trial registration information you might come to the conclusion that the advice is only appropriate for studies in the US.

That is not true.

International advice on clinical trials

The NIH provides a really useful tool for comparing the clinical research regulations from several countries:

UK trial registration

Despite these “requirements”… violations are still common as detailed in Bradley et al¹².

However, the UK has announced new infrastructure requiring 100% clinical trial registration via a collaboration between the Human Research Authority and the ISRCTN registry. See Bruckner¹³ for a thorough overview of what’s changing.

Registration to “Reduce waste”

For unexplainable reasons there is a lot written about registration of trials “reducing research waste” as it helps reduce duplication of studies.

While elements of this are true… the big takeaway is pre-registration helps prevent Questionable Research Practices.

In fact it’s the reason why GSK reached a $2.5million settlement in 2004

GlaxoSmithKline & Clinical Trial Registration

GSK chose to settle a civil case instead of engaging in an expensive legal battle over “repeated and persistent fraud” concerning the use of paroxetine in treating depression in adolescents.

Pre-registration for all studies

There’s a growing body of researchers both actually doing pre-registration and calling for it in all disciplines - particularly those that intersect with healthcare.

• Simmons et al⁷ have been running aspredicted.org since 2015 to help authors to create pre-registration reports.

• The Centre for Open Science encourages pre-registration on OSF and has in the past run a Preregistration Challenge with a monetary prize.

• I highly recommend reading Nosek et al¹⁶ which gives a great overview of pre-registration and walk through multiple examples of how it works in practice.

At the moment there’s nothing binding you to pre-register non-clinical trials but this topic is simmering away in the background

Prediction markets and replication studies

In prediction markets, investors make predictions of future events by buying shares in the outcome of the event and the market price indicates what the crowd thinks the probability of the event is.

Harvard Press Release

A final word on replicability studies

Most (if not all) of studies about replicability are deeply technical and rely on statistical methods we don’t have time to cover in this course.

This includes the foundational paper “Why Most Published Research Findings Are False” by Ioannidis¹.

Your assessment does not require you to understand or replicate any of the methodologies behind replicability studies.

Reproducibility

Let’s get back to the ASA reproducibility recommendations from Broman et al³.

Reproducibility: A study is reproducible if you can take the original data and the computer code used to analyze the data and reproduce all of the numerical findings from the study.

In order for reproducibility to be possible we need the original data to be accessible

We need Open Data.

Most papers don’t provide their data

More often than papers simply do not provide the data that they:

• Use to create charts and tables

• Use to perform statistical tests

• Use to generate their conclusions

These papers are the antithesis of reproducible.

“Data available on request”

It’s common to see the phrase “data available on request” but that’s frequently meaningless:

“Data requests to authors are successful in 27–59% of cases, whereas the request is ignored in 14–41% cases”

Tedersoo et al 2021¹⁹

Even in cases where data is returned it’s often insufficient for reproducibility, Roche et al²⁰:

• “Data” might actually be screenshots of charts stored in an Excel workbook

• “Data” might be stored in other “non-machine-readable” formats like PDF or images

• “Data” might only be provided in the post analysis form

Researchers have confusing opinions

In one of the earliest and most cited studies in data sharing in 2011 by Tenopir et al²¹ surveyed 1,329 scientists and found:

“Most respondents (at least 60% across disciplines) agree that lack of access to data generated by other researchers or institutions is a major impediment to progress in science.”

… but

“A majority of all respondents indicate they are not willing to place all of their data in central repositories with no restrictions”

These findings have been replicated again, and again.

Open Data helps everyone

Open Data vs Big Data

Big Data is great. It’s the driver behind the Internet of Things and much of modern healthcare technology.

But in most circumstances it is not Open Data.

Screenshot of opendefinition.org²⁶

Open Data short definition

Let’s minimise these definitions:

Open Data licenses

In most cases the Creative Commons’ “Choose a license tool” is the best and easiest choice if you have a dataset you want to make into Open Data.

• The CC0 license is the most permissive license.

There are special Open Data licenses used by Governmental/Charity organisations designed to waive liability for use, eg the Open Government License from the UK Government²⁷

In general it’s best to use data licenses for data and software licenses for software.

Open Data examples

There are SO MANY different sources (or publishers) of Open Data, for a good sample checkout the Open Data Essentials page from the World Bank²⁸.

Open Data and Health Data

Open Data is awesome. But if we’re responsible for data that can identify individuals or groups we have a [legal] duty of care to protect that data.

In the UK we have the Data Protection Act²⁹ which is the UK’s implementation of the General Data Protection Regulation (GDPR).

In week 6 we will discuss GDPR and the Data Protection Act in the context of anonymising data.

This isn’t a course in the law school so we won’t go hard into the definitions. But there are some things we need to discuss.

Open Data and Health Data

In the DPA²⁹ there are 6 different types of sensitive data defined in section 86

I’m highlighting the ones that cover data that you might reasonably collect and consider “health data”.

There’s some recursion, so let’s pull out the definitions of “health data” from Part 7 Section 205 and list everything together

Open Data and Health Data

Part 7 Section 205 defines the following

But we should also still include these sections from section 86:

Open Data and Health Data

There’s more reader friendly documentation about health data and DPA 2018 from the Information Commissioner’s Office³⁰.

We’ll talk about this more later.

Open Data mandates and UKRI

UKRI is responsible for the 6 UK research councils who fund most university-based research in the UK.

Some research councils have their own “data sharing policies”, but others depend on the “common principles on on research data”³¹:

Publicly funded research data are a public good and produced in the public interest. They should be made openly available with as few restrictions as possible in a timely and responsible manner.

Open Data mandates and UKRI

Understanding the whole process

Understanding the whole process (more!)

… what about code?

So far I’ve only spoken about the data element of reproducibility, we’ll get onto the code in the workshop.

Digital Object Identifiers (DOI)

DOI for more than just publications

Initially DOI were only issued by academic publishers to resolve journal articles.

Data Repositories started issuing DOI so we could resolve links to data, code and more.

Identifiers for researchers?

DOI are great for resolving our research outputs, what about uniquely identifying researchers?

Researchers often share the same names or change their names throughout their career.

General purpose repositories

• Figshare.org

  • This is my favourite repository.

  • It has a unique DOI versioning system

    • We can always get the most recent version of an object: doi.org/10.6084/m9.figshare.3761562

    • Or a specific version: doi.org/10.6084/m9.figshare.3761562.v202

• Zenodo.org

  • This infrastructure hangs off the back of CERN. If we were being pessimistic, this is the most reliable option - the others might fail.

  • DOI versioning is semantically linked - it takes a little bit more effort to obtain the most recent version of an object

• Open Science Framework - osf.io

  • The OSF tries to solve many problems at once, instead of just being a data repository.

  • OSF does not support DOI versioning.

  • The OSF interface is quite complex.

📝 Task: Setup a Collection on Figshare

SLIDE 1 OF 1

Everything we’ve spoken about has been very theoretical. I want you to go through the steps of creating a collection on Figshare.

We’re creating a Collection because it can contain multiple Figshare items. At the beginning of your research you likely don’t know exactly how many data files you’ll end up with.

Figshare for talks and more

Figshare is very much a general purpose repository.

If you create something you want to make available for the future the best thing you can do is get a DOI.

• Consider using Figshare for presentations

• Consider using Figshare for posters

We need to talk about Open Access

It used to be extremely hard to access the research that UKRI funds - despite it being funded by public money.

The Open Access movement began in the 90s and is ever growing.

It’s now a requirement of UKRI³¹ funding that

the final Version of Record or the Author’s Accepted Manuscript must be free to view and download via an online publication platform, publishers’ website, or institutional or subject repository within a maximum of 12 months of publication

… however, this often means that someone is paying an Article Processing Charge (APC).

Different routes to Open Access

Positive Publication Bias

There is a significant and very clear bias to researchers publishing “positive results”⁴¹ - which you can even see in article titles.

Publishing negative results

This is an open problem.

There was a push for new negative result journals in the early 2010s but several of these folded, including the Journal of Negative Results in Biomedicine The remaining journals have very low impact factors as they are published by smaller publishers.

In general the journals with the highest impact factors are getting better at publishing negative results.

Why are we learning all of this now?

tweetrmd::tweet_embed("https://twitter.com/charliejhadley/status/1559534088838647808?s=20&t=M0f0BvpbqLiKpDxBHiJy7w")